DOCUMENT RESUME 



ED 354 931 



JC 930 099 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



Jones Janis Cox; And Others 

Beyond Accountability: Building a Model for 

Institutional Effectiveness . 

Nov 92 

40p.; Paper presented at the Annual Conference of the 
Community College League of California (Irvine*, CA, 
November 19-21, 1992) and at the Annual Conference of 
the California Association for Institutional Research 
(Ontario, CA, November 5-6, 1992). 

Reports - Descriptive (141) — Speeches/Conference 
Papers (150) 



EDRS PRICE MF01/PC02 Plus Postage. 

DESCRIPTORS Accountability; Community Colleges; Database 

Management Systems ; ^Databases ; Data Col lect ion; 
Educational Legis lai_ion; "Educat ional Research; 
Educational Trends; Followup Studies; *Inf ormat ion 
Uti lization; Models ; ^Organizational Effectiveness ; 
Outcomes of Education; Program Evaluation; '^Research 
Methodology; School Surveys; Student Characteristics; 
Two Year Col leges 

IDENTIFIERS "California 



ABSTRACT 

In response to legislative mandates calling on 
educational institutions to become more accountable for monies spent, 
a number of community colleges in California have implemented 
research and data collection models to help them answer state and 
local questions about institutional effectiveness. However, there is 
a significant difference between providing the numerical measures to 
comply with accountability mandates and real institutional 
effectiveness. In 1983, the Los Rios Community College District 
initiated the Student Flow Research Model (SFRM) , which collected 
data on four critical areas: (1) characteristics of enrolled 
students; (2) community characteristics; (3) student performance, 
program effectiveness, and student services; and (4) activities anc* 
perceptions of transfers, graduates, and non-returning students. 
Beginning in the 1990s, each of these four data collection areas was 
significantly enhanced, and the rest of this paper focuses on the 
1990s version of the SFRM and its development into the new Collegiate 
Yearly Accountability Model ♦ In the area of community 

characteristics, a computerized Enrollment Projection Potential Model 
(EPPM) was added. To enhance data collection efforts on enrolled 
students, demographic and application data were combined with course 
and section data, allowing for a computerized program review. In the 
area of student performance, an entirely new database was developed, 
linking demographic data with transcripts. Finally, with respect to 
the former student follow-up efforts, new database changes allowed 
for results to be compiled at the program level for use by faculty. A 
discussion of five original versions or models of the EPPM is 
provided, and flow charts, data tables, and references are included. 
(PAA) 
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BEYOND ACCOUNTABILITY: 
BUILDING A MODEL FOR INSTITUTIONAL EFFECTIVENESS 



THE ERA OF ACCOUNTABILITY 



A Nation at Risk. Involvement in Learning. To Reclaim a Legacy. Integrity in the College 
Curriculum. The Neglected Majority. Time for Results. Since 1983 a steady stream of these and 
other national reports have focused on the need for educational reform. Calls for higher 
standards and tougher performance criteria have trumpeted from both within and without the 
educational establishment Something is wrong with American education, the critics charge, and 
institutional defenders have been hard pressed to meet the criticisms with anything more 
substantial than rhetoric. Faced with data that seem to show low retention rates, increased 
numbers of remedial courses, fewer transfer students, longer time to degree, and fewer course 
and program completions, educators are being asked to account to a variety of audiences as to 
how they are spending public monies and whether what they are doing is working (Coffey, 1989). 

The responses of state legislatures across the country to the calls for educational reform have 
resulted in a plethora of commission reports, resolutions, regulations and new laws concerning 
institutional effectiveness. Reforms in access, admissions, remediation, curriculum, and 
graduation standards have surfaced in many states, with concomitant reporting requirements. 
Almost everywhere, these measures represent the states' desire to improve educational quality 
by causing institutions to become more accountable fcr their outcomes (Moore, 1986). 

The move toward increased accountability at the state level has also introduced the potential 
for— and the reality of— performance-based funding of educational institutions as a logical 
corollary. Competitive incentive grants, withholding of percentages of state appropriations, 
increased percentages of funding based on meeting performance criteria, are all in existence in 
various states across the country. Such performance-based systems certainly put teeth into state- 
level mandates for accountability, but also raise crucial questions of who is to determine the 
evaluation criteria and appropriate performance measures. Whether it be legislative staff, state 
board members, or institutionally-based task forces, wise choices in evaluation and performance 
measures are essential to real educational improvement— and potentially even to institutional 
survival. 



In addition to the state and federal mandates for increased accountability, the regional 
accreditation agencies have instituted their own calls for more accountability through new 
accrediting standards. Beginning in 1984 with the Commission on Colleges of the Southern 
Association of Colleges and Schools, the regional agencies have written standards that established 
the evaluation of effectiveness as a major criterion for accreditation. In 1988, the Western 
Association of Schools and Colleges adopted iu» standard on "Institutional Effectiveness" that 
calls for each institution to initiate a research program that assesses student skills, performance 
and outcomes, and links that information to the institution's planning process (WASC, 1988, 
p. 16). 



The California Picture 



In California, the call for increased accountability for the state's community colleges and its 
linkage to institutional research is perhaps nowhere better stated than by the Commission for the 
Review of the Master Plan for Higher Education in its study The Challenge of Change- A 
Reassessment of the California Community Colleges (1986, p. 12): 

Institutional research is essential to determine which types of programs work best 
with which students under what circumstances, and to ensure the wisest use of 
public funds in meeting student and community needs. There is relatively little 
statewide institutional research available to evaluate the effectiveness of the 
Community College transfer, vocational, or remedial prjgrams, which are of 
particular concern to this Commission. If these programs are to be implemented 
successfully and cost-effectively, they must be accompanied by research and 
evaluation from the start, to strengthen these programs as they develop as well as 
to evaluate their ultimate merit. Significant additional funds will be needed for 
this research. 



To respond to the challenge of accountability, institutions have begun to identify and implement 
research and data collection models that will help them answer both state and local questions 
about institutional effectiveness. What has become apparent over the past several years, however, 
is that there is a significant difference between accountability— which has come to be focused 
on data elements and numerical measures (what used to be called "compliance"V- and real 
institutional effectiveness. With the advent of California's community college reform bill 
(generally known as "AB 1725"), and the passage of legislation detailing the state's new 
accountability system for the community colleges, colleges across the state have added research 
offices to help meet the demands for accountability data to be entered into the new statewide 
Management Information System (see Coffey and Hamre, 1988). There it will be analyzed by 
the State Chancellor's Office staff to help answer the various state agency and public questions 
about how well the community colleges are responding to one or another mission or mandate. 
Clearly, the era of accountability is upon us. The remainder of this paper will describe how one 
district, often noted for its successes in proving what it's doing is working, has moved beyond 
accountability ana built a real model for institutional effectiveness. 

IN THE BEGINNING: THE STUDENT FLOW RESEARCH MODEL 

In 1983, considerably before the current emphasis on accountability, the Los Rios Community 
College District began to build what would later come to be known as the "Student Flow 
Research Model." Given the district's major emphasis on using research results for planning and 
decision-making, we sought a model that would bring together information in four critical areas- 
(1) Community: characteristics of the district's service area population; (2) Enrolled Students- 
characteristics of the entering stud-nt population; (3) College Experience: evaluation of student 
performance, programs and services; and (4) Former Student Follow-Up: follow-up of transfer 
and occupational graduates, and nonretuming students (Coffey, 1987). (See Figure 1.) 
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Rather than focus on data to be collected, we intended that the research produced in these four 
areas would answer a series of seven key questions, questions we thought any district or college 
should be able to answer: 

1. What is our community like and who are our potential students? 

2. Who are our enrolled students? Do they differ by college? Do they reflect the 
community at large? 

3. What kinds of preparation do our students bring to our institutions? Are they prepared 
for our college-level classes or do they need remediation? 

4. What are the educational goals of our students and do these goals differ by age, sex, 
ethnicity, work status, or economic level? 

5. How well are we meeting our students' needs? Is what we're doing working and how 
do we know? 

6. What happens to our students once they leave? Are they successful as transfers to 
four-year institutions? In finding jobs? In improving skills and potential if currently 
employed? 

7. Finally, how can we improve what we're doing? 

While we had originally thought that these key questions were almost self-evident, the 
considerable interest in the model and questions from California and across the country indicated 
that the concept of building a research model around student and institutional effectiveness 
questions was quite new. The basic components of each of the four areas of the model are 
shown in Figure 2. 

For ihe Community information, we developed district and college service area demographics, 
based on the most recent Federal Census, and "geocoded" to our service areas based on a ZIP 
code analysis of our enrolled students. Labor market information from a variety of sources is 
included in the Community information, as is economic data about particular communities. For 
the Enrolled Students information, we largely use student demographic, goal and financial aid 
status data from the application data base, along with course and unit load information from the 
course and program data base. 

The College Experience area was the one in which we found the least information available when 
we designed the Student Flow Research Model, because the studies of student preparation, 
persistence and performance had yet to be done. We knew even then, however, that we wanted 
to do internal evaluations of our programs and services according to the "three EV: equity, 
efficiency and effectiveness. To do truly effective program and service evaluations, we knew we 
needed some means of tying student demographic and goal data to course-taking patterns and 
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transcripts— and in the early 1980s, without the kinds of relational data bases that are now in use, 
we had no way to do it 

Finally, the information on our Former Students was becoming available, based on our Measures 
of Progress studies (Lee, 1987, 1991), which used survey research to follow what had happened 
to our students once they h^d left our colleges. Indeed, our research in that area was used in 
pivotal legislative discussions about the supposed high community college drop-out rate, when 
we were able to document that many of our students came for job upgrading and retraining 
opportunities that required only particular courses rather than degrees; in short, we documented 
the phenomenon of the "drop-in" student 

Over the past nine years, we have finetuned the Student Flow Research Model and kept our eyes 
on the increasing importance of those seven key questions. And we were able to add, through 
the addition of some unique data base developments, the ability to answer the questions of what 
was happening to our students while they were in our institutions. The next section of this paper 
will focus on our 1990s' version of our Student Flow Research Model and its development into 
the Collegiate Yearly Accountability Model— or our CYA model, for short (and yes, the allusion 
to the more common meaning of those initials is intended). 

THE NEW MODEL AND WHAT MAKES IT WORK 

Figure 3 illustrates the new data base developments that make the new model work. In the 
Community information area, we have added the Enrollment Potential Projection Model 0 or 
EPPM**, an entirely unique computerized enrollment projection model that has recently been 
approved by the state Department of Finance for use in projecting the enrollments for Folsom 
Lake College, the district's proposed new college (usually, only DOF can do such projections). 
In the Enrolled Students information area, we have combined the demographic and application 
data from the original model (our "student master files") with course and section data (our 
"master schedule files"). This has enabled us to do computerized program review analyses thr.t 
include five-year demographic, enrollment, credit load, and day/evening trend information. In 
the College Experience area, we have again constructed an entirely new data base— the Student 
Performance Data Base or SPDB— that links the student demographic and course data with 
performance data (transcripts) into a "quasi-relational" data base, allowing us to do computerized 
transcript analysis for any population of students quickly and accurately. Finally, in the area of 
Former Student Follow-Up, we have enhanced our extensive student follow-up survey data base 
through the ability to compile the results not only at the district or college level, but also at the 
program level for use by faculty doing program planning and review. The next sections will 
cover each of these areas in greater detail. 

The Enrollment Potential Projection Model 

The Los Rios Community College District, as part of its new strategic planning process, began 
in early 1991 to update and enhance the enrollment projection model first developed by Dr. Marc 
Hall for the 1988 facilities project. The new model developed by the district's Office of Planning 
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and Research is called the Enrollment Potential Projection Model (EPPM) and is quite different 
from the original, dnc to its capability of producing alternative projection scenarios. This is 
particularly important in a multi-college district with a variety of outreach center locations. The 
computerized EPPM has proved useful in the district's strategic planning discussions, and 
particularly in helping to answer major policy questions about the communities our current 
colleges and centers serve, the demographic changes projected for those communities, the 
potential effects on the size and nature of our future student populations, and the ways in which 
the district can address the dramatic growth in the Sacramento region. The EPPM provides the 
district and colleges with capabilities crucial to comprehensive and cost-effective planning: 

(1) The ability to project the potential for increased enrollments in the district and at the 
colleges based on the most accurate data available; 

(2) The ability to project the results of a variety of alternative planning scenarios, including 
differences in community area growth rates and college participation rates; and 

(3) The ability to project the potential impacts on district enrollments of proposed new 
colleges or centers. 



The EPPM was designed to help the district and colleges answer the key planning questions 
facing any district as it examines alternatives to handle projected growth: 

• How many students will we need to serve in the coming years? 

• From which community areas of the district's service region will these students come? 

• What impacts might the differential rates of growth throughout the district's service area 
have on projected district and college enrollments? 

• What impacts might target marketing to particular areas have on potential district and 
college enrollments? 

• What are the available options for meeting the increased demand, and what are the 
potential impacts of each option on the current colleges and centers? 

• Are there enrollment limits that we wish to set for current or projected colleges or centers, 
and what could be the potential impacts of such limits? 



The Enrollment Potential Projection Model projects potential enrollments for the Los Rios district 
and the individual colleges based on current enrollment participation rates, using actual and 
forecasted population projections based on 1990 Census Data and governmental population 
projections. Figure 4 on the next page illustrates the basic components of the model. 
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The forecasted population projections for Sacramento and Yolo County were provided by the 
Sacramento Area Council of Governments (SACOG). El Dorado County population projections 
were also supplied by SACOG, but were derived from Department of Finance projections. 

The EPPM divides the Los Rios district into 24 individual community areas. Population 
projections for each of the community areas were supplied by SACOG for 1990, 1995 and the 
year 2000. Actual 1990 enrollments were derived through analysis of the district's Fall 1990 
First Census Student Master Files. Enrollments were identified for each community area, and 
broken out by college (including the Placerville Center). 

Participation rates of students from each of the community areas were derived by the following 
formula: 6 



1990 Fall Student Enrollment 
from Community Area 



1990 Population 
from Community Area 



Participation Rate 



Projected enrollments for any community area for the years 1995 and 2000 were computed by applying 
the following formula: 6 



1990 Participation Rate 
by Community Area 



Population of 
Community Area 
in Year of Projection 



Estimate of Student 
Enrollment in Year of 
Pro;*sction by 
Community Area 



Individual college enrollments were computed by calculating attendance distributions by college using the 
following formula: 6 



Fall 1990 Total College 
Student Enrollment from 
Community Area 

Fall 1990 Total District 
Student Enrollment from 
Community Area 



Ratio for 

Each 

College 



Estimate of 
Student Enrollment 
in year of Projection = 
by Community Area 



Estimate of 
Student Enrollment 
by College by 
Area in Year of 
projection 



There are five original , versions of the Enrollment Potential Projection Model*. Model I, the 
most basic model is patterned after the model developed by Dr. Marc Hall (Los Rios Community 
College District Student Enrollment Analysis and Projections, Report II; Hall and Coffey, 1988) 
The first page of the model is essentially the "data page" where the parameter are set. It shows 
the base population projections for each community area; the 1990 student population that 
attended from each community; the actual 1990 sti ent participation rate (the percentage the 
student population is of each community area); the 1995 projected community area populations; 
the resulting projected 1995 student population (based on holding the actual 1990 participation 
rate steady); the year 2000 projected community area populations; and the resulting projected 
year 2000 district student population (again, based on the 1990 participation rate). 
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Summary reports for 1995 and the year 2000 are provided on the following pages of the model 
showing projected potential enrollment distributions for the individual community areas for each 
of the three current colleges in the district, and for the Placerville Center, for 1995 and 2000. 
Totals printed at the bottom of each page show district projected potential enrollments. 

Model XI )uilds on the foundation of the first model, but allows for user-adjustable 
participatic i rates for each community area. Since participation rates can be increased by 
proximity to a center or college, as well as by marketing, the ability to adjust the participation 
rate is an important part of the model. The same summary reports for 1995 and the year 2000 
are also included in this model. 

Model in, a more complex model, also provides for user-adjustable participation rates by 
community area, but in addition auows for adjustable population multipliers for each 
community area. This permits adjustment of the community area populations if more current 
population projections are available or other factors suggest changes. In addition, Model m 
allows the user to set enrollment limits for each college, then permits the "excess 1 ' enrollments 
to be allocated to a "new' 1 college if forecasted enrollments for the current college exceed the 
user-defined limits. The same summary reports for 1995 and the year 2000 are also included in 
this model. 

A crucial component of the EPPM is the ability to reflect the enrollment limits established for 
each of the districts current colleges and the Placerville Center, so that "excess" enrollments can 
be applied to the "new" campus in the model. Model IV provides for setting of college 
enrollment limits. The Chancellor and the college presidents agreed upon the college enrollment 
limits to be used in the EPPM. These limits were established after consideration of State facility 
usage standards, maximum buildout for each of the current campuses based on the district's 1991 
Capitol Ouday Plan, and the ability to meet recognized and generally accepted levels of service. 
American River College's enrollment limit was set at 21,000; Cosumnes River College was set 
at 10,000; the Placerville Center was set at 3,000; and Sacramento City College was set at 
18,000. The EPPM was modified to reflect the mutually agreed-upon enrollment limits, and then 
was run to produce the "Model IV" baseline projection. 

Model V, the model ultimately used and approved by the Department of Finance as the formal 
projection for the new Folsom Lake College enrollments, adds one additional feature to 
Model IV: a "Special Report" that identifies the proportions of people most likely to attend 
the new college from each community area, based on proximity, on analysis of student 
enrollment by ZIP code, and on responses to a community survey. While Model IV projects all 
the potential new students coming to the district, the Model V Special Report recognizes that not 
all the new students from throughout the district will attend the new campus, and instead reflects 
those most likely to attend. 

The EPPM has provided a unique means of answering the key questions of what our various 
community areas are like and what level of potential future enrollments we will need to serve. 
The best example of the effectiveness of this new research model has been its approval by the 
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state Department of Finance, the state Chancellor's Office, and the California Postsecondary 
Education Commission as the basis for the formal enrollment projections for the Los Rios 
district's fourth college. The model is highly flexible and permits us to analyze any number of 
alternative scenarios, based on adjustable participation rates, variable population multipliers, 
different college enrollment limits, and the likelihood of people from different community areas 
attending each of our colleges. The EPPM has generated a great deal of interest from other 
community college districts around the state as we all seek to understand how we can plan to 
meet the increasing enrollment demands in face of continuing fiscal stringency. 

Combining Student and Course Master Files 

In the area of Enrolled Students mentioned above, we have now combined our student application 
and demographic files with our master schedule files, in order to be able to analyze historical 
student enrollment data and its relationship to changes in course-taking patterns. Figure 5 on the 
next page illustrates the linkages of these files and the kinds of information available. 

Our Student Master Files (SMF) contain studen?: application and demographic information 
collected during the registration process. Information such as educational goal, full- or part-time 
status and major are some of the application items that are collected. Demographic information 
includes such elements as age, gender and ethnicity which are also collected during the 
application registration process. 

Our Master Schedule Files (MSF) contain student ,.urse enrollment information collected from 
the students during the registration process. The actual class schedules, which include course, 
section, day and time and instructor information, arc stored in these files. 

All information collected from students during the registration process is stored on the district's 
mainframe computer(s). The district's research department routinely extracts the information 
from the Student Master Files and Master Schedule Files stored on the district's mainframes at 
the end of Fourth Week Census. These files are then restructured into an SPSS-readable format, 
thus simplifying analysis. 

The ability to link these two types of quite extensive mainframe files using SPSS-X thus became 
our first look into how student enrollments both affect and reflect course planning decisions. At 
the college level, having the five-year history of key program trends relieved the faculty from 
having to search for and analyze such information as they had done in the past, permitting them 
to spend their time instead on evaluation of the implications of the trends for planning. Having 
solid and visual evidence of the various program trends also made it easier for faculty and 
administration alike to identify those programs that were weak in enrollments and in need of 
improvement or potential deletion. This helps make the program review process at least 
somewhat less threatening, and considerably less adversarial. 
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Student/Course Files 



Student Master 
File 

* Demographics 

* Application 

* Goal 
Major 



Master Schedule 
File 

* Course Title 

* Section 

* Instructor 

* Contact Hours 




Answers the Following Questions 

Who are our Students? 
What Courses are they taking? 

Outcomes 
Program Review 

BA Degree Analysis 



FIGURE 5 
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The ability to link Student Master Files and Master Schedule Files has also proven invaluable 
for our recent analysis of baccalaureate students' course-taking patterns and demographics. New 
state legislation passed during the 1992 fiscal crisis mandates a differential fee of $40 per unit 
(in addition to the usual $10) for those community college students who already have a 
baccalaureate or higher degree. In order to plan for the Spring 1993 semester, when the new fee 
takes effect, the district needed to answer two key questions: 

1. What courses are these baccalaureate degree students enrolled in? 

2. What do we know about these students demographically and in terms of their reasons 
for enrolling? 

To answer these questions, we analyzed the entire Fall 1992 Student Master File to determine 
the students' goals and demographics, then combined the information with the course information 
for these students contained on the Fall 1992 Master Schedule File. The resulting analysis 
provided a profile of the baccalaureate students districtwide: the majority were women, most 
over the age of 30, predominantly white, who had enrolled primarily for occupational reasons. 
Each college was able to see its own baccalaureate student profile, and was p ided an analysis 
of the courses these students were taking, including frequency of course enroL cnts by academic 
department. The colleges used this information to plan their Spring course schedules and to 
anticipate the areas which might be hardest hit by enrollment losses due to the fee. 

Finally, the list of baccalaureate students from the Student Master File was used as a mailing list 
for a special survey which was sent to each of these students to elicit additional information 
about their plans for the Spring 1993 semester and their reactions to the new baccalaureate fee. 

The Student Performance Data Base 

Perhaps the most critical new addition to our research capabilities in terms of evaluating 
institutional effectiveness is the new Student Performance Data Base. When we began to develop 
the SPDB, the district's student information was designed such that the demographic and 
application information was stored in "flat file" structure on the Honeywell mainframe computer, 
while the transcript information was stored in a newly purchased relational data base format on 
the district's CDC Cyber mainframe. The statistical package SPSS-X was installed on the Cyber. 
Linking files across these two platforms with two entirely different data base structures would 
be a complicated process to say the least And yet we knew that with demands for accountability 
increasing, we needed to look at data in new ways to answer the key questions on student 
preparation, persistence and performance. We decided to move ahead, creating our own quasi- 
relational data base that would link all the disparate files into a usable, effective structure. 

We needed to be able to answer the key questions about what was happening to students while 
they were in our three colleges, whether they were persisting and performing successfully, and 
how performance might differ for different groups of students in different programs. To do such 
analyses we needed the capability of essentially examining transcripts as would a counselor, 
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looking not only at overall or course-specific GPA, but at the kinds of courses the student was 
taking — remedial, transfer or occupational — since student preparation for college-level work is 
a key indicator of student performance, and an important issue in community colleges. 

Figure 6 illustrates the major components of the Student Performance Data Base. Essentially, 
student application, demographic and transcript data files on the mainframe computer were linked 
using student Social Security numbers and the file handling capabilities of SPSS-X. Converting 
the transcript data files proved the most difficult, since the on-line viewing capabilities designed 
for the counselors at each college to view individual student transcripts meant that the relational 
data base structure was highly complex. We decided that duplicating the transcript data base in 
SPSS-readable format was a poor idea since that data base contained over four million transcript 
records, with over two billion characters. We developed a means of using the Cyber' s fourth- 
generation query language to navigate the transcript data base, extract the necessary files on a 
project-by-project basis, and build those files into ar. SPSS-readable format for each project as 
needed. Once a study was completed, the SPSS-ieadable transcript information would be purged 
from the system, or kept in an off-line format. 

In order to produce computerized transcript analyses for particular types of courses, we had to 
design a process to identify every course on the district's data base as to type — whether the 
course is a general education or an occupational education course, and at the transfer, AA degree 
or remedial level. In addition, we wanted to be able to select or aggregate the information at the 
department or even course level, so as to answer such questions as how students who take 
remedial English courses perform in courses at the Freshman Composition level, or what the 
GPAs of student athletes are when figured with and without PE courses. This required having 
the district's Research Coordinator (who also spent 14 years in the district as a counselor), 
identify each course as to type, design the program and then build a data base of twenty years 
of district course offerings (all courses ever taught in the district over that time and therefore on 
historical transcripts). We could finally classify each course a student had attempted over the 
past twenty years, taking into consideration renaming of courses and changes in classifications 
over time. Computerized transcript analysis was now a reality. 

Because the SPDB also includes the student application and demographic data, performance 
analyses can be done along demographic lines or according to student goal or major, and student 
populations can include any group of students identifiable by Social Security number — those who 
have matriculated* those on financial aid, those in special programs or receiving special services. 
This permits true analyses of the effectiveness of programs and services on particular populations, 
and pre- or co-requisites on student performance in subsequent courses. The information for 
, specific populations in specific studies can be requested by staff at our colleges and downloaded 
if* any of a variety of formats for use in microcomputer SPSS analysis. Recently, the college 
research offices have begun designing their own research data bases, using data downloads from 
the SPDB that are done for all students at each college each semester. Agreement on the most 
critical data elements was reached by the college research offices so that the full downloads can 
be done once at the district level and then disaggregated by college. Figure 7 shows a flowchart 
of the key elements of the Student Performance Data Ba.*e 
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TheSPDB has been used in a wide variety of studies since its inception. Perhaps best-known 
is the district s Moving On: A Cooperative Study of Student Transfer (Jones, 1991), done with 
the University of California at Davis and with California State University at Sacramento This 
study, which has been published nationally, went far beyond the usual counts of transfer students 
1°J iXT* u 11 leV ? ° f P"*^ 00 ' Persistence and performance at the community college 
and then how they performed m terms of units attempted and completed, CPA and baccalaureate 
degree attainment at each of the universities. We were able to identify those students who had 
uiken remedial courses in English, math or ESL at the community college level and then to track 
the remedial vs. non-remedial cohorts into and through the universities to determine the effects 
that remediation had on subsequent performance. Analyses were performed along demographic 
hues for various ethnic groups and for both genders. The findings-that the community college 
transfers performed as well as or better than the university native students, and that the remedial 
cohort graduated from the universities in proportions not much dL/erent from the non-remedial 

£nS~TT f ha S, d aCr °/ S *"* SegmentS and hdped t0 identif y P rofUes of the suc <*ssful 
muisfer student. This information is being used by counselors, tran fer center directors and 
others to assist students in developing and reaching their transfer goals 

More recently, the SPDB has been used in an analysis of our baccalaureate degree-holding 
students in terms of their demographics and course-taking patterns. This is particularly important 
in light of the new $50 per unit fee that the Legislature mandated for all such students in the 
community colleges. We were able to identify those students who held the baccalaureate degree 
(according to their declaration on the application), and provide a picture of who these students 
were and what courses they were enrolled in, to help in planning next Spring's course offerings 
since many of these students may not be back in January. The fact that these students were 
predominantly older women, almost half of whom enrolled with specific job attainment 
upgrading or retraining intentions (and a considerable percentage with transfer intentions) is 
helping us focus attention on how to meet the needs of these students-who will now pay 
university-level prices for their community college courses. In addition, the district's planning 
and research office has designed and mailed a survey to these students to determine the impact 
of the differential fee on their educational intentions. 

Now in the planning stages is an evaluation of the new Math, Engineering and Science 
Achievement/Community College Program in the district's three coUeges^unJed wiTnew tZ 
appropriations which aims to increase the numbers of underrepresented transfer students in math- 
and science-related fields. The SPDB will enable us to identify and then follow these targeted 
students to monitor their persistence and performance as they receive the program's various 
special services as well as determining how many such students achieve eligibility for transfer 
The ability of the SPDB to track students at the department or even course level will be crucial 
to the success of the evaluation of the effectiveness of MESA/CCP. 
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Former Student Follow-Up Data Base 



The Los Rios district began its student follow-up efforts with a pilot survey in Spring 1983, long 
before the demand for accountability became an issue of state and national concern. We wanted 
to know what happened to our students as a result of their community college experience: Were 
they employed? Did they transfer to another college or university? Had their goals been met? 
(Figure 8). 

The Spring 1983 pilot study involved only "occupational" students, following guidelines required 
for Vocational Education Data System (VEDS) reporting. As a result of that study, Los Rios 
decided to expand its survey population to include all graduates and certificate earners (general 
education as well as occupational majors), plus the cohort of nonreturning nongraduates (referred 
to as NRSs) selected according to VEDS guidelines. Since that study also showed that the VEDS 
guidelines misidentified from 20% to 40% of the students' major programs, Los Rios decided 
to use its own record of the degrees/certificates actually awarded (if any). 

The Los Rios Student Follow-Up Data Base now includes responses from almost 12,500 former 
students who attended the district's three colleges between 1982 and 1990 (Figure 9). Survey 
results, reported in the district's Measures of Progress series, are aggregated on the basis of the 
respondents' self-reported educational goals: University Transfer, Occupational Preparation, 
Occupational Retraining, and Personal Interest, with a separate analysis of responses from NRSs 
(regardless of goal). The results are also summarized by program, by college, and for the district 
as a whole. 

The early Los Rios studies were conducted annually (1984 through 1987), and were compared 
in Measures of Progress: A Four-Year Retrospective (Lee, 1987). As recommended in that 
report, subsequent studies have been conducted biennially (but still include all students who 
would have been included in the annual surveys). Measures of Progress: Student Follow-Up, 
1984-1991 (Lee, 1991) discussed the series of studies in a newsletter format, identifying trends 
and summarizing the responses received during that eight-year period. 

In addition to collecting the employment data required for reporting on occupational education 
students, the Los Rios questionnaire elicits information on the students' primary reason for 
enrolling at the community college, their current educational status (including information on 
their transfer school, if any), and their ratings of community college courses completed and 
services used (Figure 10). Thus the surveys go far beyond "compliance" reporting and provide 
considerable additional information useful in college and district planning. 

Survey procedures have been modified over the eight-year period, usually in direct response to 
suggestions or requests from users of the data. One of the most important of these changes, 
made at the request of college deans of occupational information, was the development of 
program-specific summaries which are prepared for each major program having ten or more 
responses. Each summary shows the number and percentage of respondents answering each of 
the questionnaire items, as well as basic information regarding that program's survey population 
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(number of graduates, certificate earners and NRSs; adjusted response rate; and demographic 
profiles of the respondents). This change of format and the breakdown of information by 
program makes the information much more useful to college staff in their curriculum and 
program review efforts and in accreditation sfif-study reports. 

Another refinement — the designing of individual skills-specific questionnaires for various 
occupational majors — resulted from Los Rios* involvement in the early stages of what is now 
called the Statewide Student Follow-Up System (SSFS): faculty at each of our colleges used 
drafts developed for that earlier project to design supplementary forms tailored specifically to Los 
Rios occupational programs they wanted to examine more closely. Currently, in cooperation with 
the SSFS project manager, the basic questionnaire to be used for the Spring 1993 Los Rios 
student follow-up survey is being revised to ensure that Los Rios can participate in the SSFS 
study while continuing to gather the extensive additional data our colleges have come to expect. 

The Student Follow-Up Data Base enabled the district to comply in a timely manner with the 
Student Right-to-Know regulations, since we already had on hand a large amount of historical 
information regarding where former students found employment and the relationship of their jobs 
to their community college education. The data base is also expected to be very useful in 
meeting the new VATEA reporting requirements. 

Information from the data base has also been used in a variety of other ways: by the colleges, 
to provide information for their Occupational Education Advisory Committees; by members of 
Los Rios' Board of Trustees and the Director of Community and Media Relations, in responding 
to business/community concerns; in conference presentations and published articles on community 
college accountability at both state and national levels. 

Los Rios continues to refine its student follow-up system with each new biennial survey, in oider 
to continue to meet the need for even more comprehensive answers to the basic question: What 
happened to our students as a result of their community college experience? The enhancements 
described above and others in the planning stages will maintain the Student Follow-Up Data Base 
as a dynamic and essential component of the district's accountability model. 

FROM ACCOUNTABILITY TO EFFECTIVENESS 

The genesis from the Student Flow Research Model to the new CYA Institutional Effectiveness 
Model has taken almost ten years. Much trial and error has been involved, along with a strong 
focus on the original key questions from the Student Flow Model. Those questions are still the 
ones we need to answer, and our ability to examine our programs and services in terms of their 
effectiveness in producing successful students is now more important than ever. In this era of 
increasing student enrollment demand and diminishing resources, we no longer have the luxury 
of maintaining programs and services that are not meeting student needs. And more than ever 
before, we are being asked to demonstrate — to the Legislature and the Governor, to state and 
federal agencies, to our boards, to our communities, to our faculty and to our cuiTent and 
potential students — that what we are doing is working. 
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In designing our model, we kept focused on the questions to be answered rather than on the data 
to be collected; that was true in the original Student Flow Research Model and it continues to 
be the best process for enhancing the model. Focusing on what happens to our students as they 
move into, through and beyond our colleges has helped us identify v-hich pieces of the model 
we needed to add next — and helped us to work around seemingly impossible barriers creating 
what we needed as the technology permitted it (and sometimes when it didn't). As we were able 
to get to finer levels of analysis (course-taking patterns; the effects of remedial courses; GPA 
with and without PE courses), we have been better able to identify where our programs and 
services are having an impact and where they are not Figure 1 1 summarizes the changes to the 
model over time, and Figure 12 illustrates some of the key outcomes we have achieved — the 
move from compliance and accountability to institutional effectiveness. 

Perhaps most of all, we have learned how complex a process is analyzing institutional 
effectiveness — and sometimes, how political. Documenting increases in the number of transfers 
a community college is sending to public four-year universities is one thing; finding out what 
proportion of those students ever received their baccalaureate degrees, and how it differed by 
gender and ethnic group, is quite another. V'hile it is tempting to hold individual colleges 
accountable for particular outcomes (numbers who transfer; students who graduate), it is also true 
that colleges do not have control over all the factors that affect student performance, and that 
students themselves must take some responsibility for their success or failure. However, as 
researchers who believe that the only real reason for research in the community colleges is to 
help improve our institutions on behalf of our students, and to determine which programs work 
best with which students under what circumstances, we believe that the more complete the 
information, the more compelling an argument for change. Moving from compliance and 
accountability to true analyses of institutional effectiveness is difficult and time-consuming. It 
may be the most we can do; it is the least our students deserve. 



*************** 
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